1. Field of the Invention
The present invention relates to an information processing method and an information processing apparatus that use information indicating the image coordinates in a captured image of indices arranged in the physical space to calculate the position and/or orientation of a camera viewpoint.
2. Description of the Related Art
Mixed reality has been much researched in recent years for the purpose of realizing seamless combination of the physical space and a virtual space. An image display apparatus presenting the mixed reality adopts a video-see-through technology in which an image of a virtual space (for example, a virtual object or characteristic information drawn by computer graphics) generated in accordance with the position and orientation of the viewpoint of an image capture apparatus, such as a video camera, is superimposed on the image in the physical space, captured by the image capture apparatus, and the superimposed image is displayed.
Such image display apparatuses are expected to be applied to new fields different from known virtual reality. The fields includes an operation aid superimposing an image inside the body of a patient on the body surface and displaying the superimposed image and a mixed reality game in which an operator battles against a virtual enemy floating in the physical space.
Such an application commonly requires accurate registration of the physical space with the virtual space and many efforts have been made hitherto in order to achieve accurate registration. Problems about the registration in the mixed reality are involved in the calculation of the position and orientation of the viewpoint in the physical space, that is, in a world coordinate system (a coordinate system in which one point in the physical space is defined as the origin and three axes orthogonal to each other are defined as an X axis, a Y axis, and a Z axis).
Generally, in order to solve these problems, a plurality of indices are arranged in the physical space, and the coordinates in the world coordinate system of the indices and the coordinates of projected images of the indices in an image captured by an image capture apparatus are used to calculate the position and orientation of the viewpoint in the physical space. Calculation methods in, for example, photogrammetry have been suggested (see R. M. Haralick, C. Lee, K. Ottenberg, and M. Nolle, “Review and analysis of solutions of the three point perspective pose estimation problem,” International Journal of Computer Vision, Vol. 13, No. 3, pp. 331-356, 1994 and D. G. Lowe, “Fitting parameterized three-dimensional models to images,” IEEE Transactions on PAMI, Vol. 13, No. 5, pp. 441-450, 1991).
In the above methods, it is required that the accurate coordinate values of the indices in the world coordinate system are known. That is, it is necessary to measure (calibrate) the accurate coordinate values of the indices in the world coordinate system by a certain method. Accordingly, the coordinate values are generally measured by using a ruler or survey instrument prior to the calculation of the position and orientation of the viewpoint or measured in advance by using a calibration tool disclosed in, for example, G. Baratoff, A. Neubeck, and H. Regenbrecht, “Interactive multi-marker calibration for augmented reality applications,” Proc. ISMAR2002, pp. 107-116, 2002.
However, there is a problem in that the indices should not be moved in the calculation of the position and orientation of the viewpoint because known calibration methods are performed before the calculation of the position and orientation of the viewpoint.
Since objects (for example, the hands of a user or objects operated by the user) other than the indices exist in the physical space and the positions of the objects can be shifted, the objects can block the space between the indices arranged in the physical space and the viewpoint in some cases to interfere the calculation of the position and orientation of the viewpoint. In such cases, there may be a need to move the indices shaded behind the objects.
In order to resolve this problem, moving indices are suggested. For example, Yasue Kishino, Masahiko Tsukamoto, Hiroshi Sakane, Shojiro Nishio, “A Translucent Visual Marker with Autonomous Mobility on Computer Display”, Proc. Interaction 2002, Information Processing Society of Japan, Vol. 2002, No. 7, pp. 14-21, March 2002 (hereinafter Yasue Kishino et al.) discloses a method of realizing indices that are displayed in the screen of a computer and that is capable of autonomous movement. However, with the method disclosed in Yasue Kishino et al., the indices are only movable and it is not possible to calculate the positions in the world coordinate system of the indices after the movement. Accordingly, it may be impossible to use such a method in order to calculate the position and orientation of the viewpoint in the world coordinate system.